{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用fuzz testing模块测试模型安全性\n",
    "作者：MindSpore团队、[徐威](https://gitee.com/chow-chow)  \n",
    "`Linux` `Windows` `Ascend` `GPU` `CPU` `模型评测` `企业` `高级`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述 \n",
    "传统软件的决策逻辑由代码逻辑决定，传统软件通过代码行覆盖率来判断当前测试是否充分，理想情况下覆盖率越高，代码测试越充分。然而，对于深度神经网络而言，程序的决策逻辑由训练数据、网络模型结构和参数通过某种黑盒机制决定，代码行覆盖率已不足以评估测试的充分性。需要根据深度网络的特点选择更为适合的测试评价准则，指导神经网络进行更为充分的测试，发现更多的边缘错误用例，从而确保模型的通用性、鲁棒性。  \n",
    "\n",
    "MindArmour的fuzz_testing模块以神经元覆盖率作为测试评价准则。神经元覆盖率，是指通过一组输入观察到的、激活的神经元数量和神经元输出值的范围。我们通过神经元覆盖率来指导输入变异，让输入能够激活更多的神经元，神经元值的分布范围更广，从而探索不同类型的模型输出结果、错误行为。  \n",
    "\n",
    "这里以LeNet模型，MNIST数据集为例，说明如何使用Fuzzer。\n",
    "\n",
    ">本例面向CPU、GPU、Ascend 910 AI处理器，样例代码：<https://gitee.com/mindspore/mindarmour/blob/master/examples/ai_fuzzer/lenet5_mnist_fuzzing.py>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实现阶段\n",
    "### 导入需要的库文件\n",
    "下列是我们需要的公共模块、MindSpore相关模块和fuzz_testing特性模块，以及配置日志标签和日志等级。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "\n",
    "from mindspore import dataset as ds\n",
    "from mindspore import dtype as mstype\n",
    "import mindspore.dataset.vision.c_transforms as CV\n",
    "import mindspore.dataset.transforms.c_transforms as C\n",
    "from mindspore.dataset.vision import Inter\n",
    "import mindspore.nn as nn\n",
    "from mindspore.nn import SoftmaxCrossEntropyWithLogits\n",
    "from mindspore.common.initializer import TruncatedNormal\n",
    "from mindspore import Model, Tensor, context\n",
    "from mindspore.train.callback import LossMonitor\n",
    "\n",
    "from mindarmour.fuzz_testing import Fuzzer\n",
    "from mindarmour.fuzz_testing import ModelCoverageMetrics\n",
    "from mindarmour.utils.logger import LogUtil\n",
    "\n",
    "\n",
    "LOGGER = LogUtil.get_instance()\n",
    "TAG = 'Fuzz_testing'\n",
    "LOGGER.set_level('INFO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 参数配置\n",
    "配置必要的信息，包括环境信息、执行的模式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "context.set_context(mode=context.GRAPH_MODE, device_target=\"GPU\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "详细的接口配置信息，请参见`context.set_context`接口说明。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 运用Fuzz Testing\n",
    "1. 建立LeNet模型\n",
    "- 加载MNIST数据集：利用MindSpore的dataset提供的`MnistDataset`接口加载MNIST数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate dataset for train of test\n",
    "def generate_mnist_dataset(data_path, batch_size=32, repeat_size=1,\n",
    "                           num_parallel_workers=1, sparse=True):\n",
    "    \"\"\"\n",
    "    create dataset for training or testing\n",
    "    \"\"\"\n",
    "    # define dataset\n",
    "    ds1 = ds.MnistDataset(data_path)\n",
    "\n",
    "    # define operation parameters\n",
    "    resize_height, resize_width = 32, 32\n",
    "    rescale = 1.0 / 255.0\n",
    "    shift = 0.0\n",
    "\n",
    "    # define map operations\n",
    "    resize_op = CV.Resize((resize_height, resize_width),\n",
    "                          interpolation=Inter.LINEAR)\n",
    "    rescale_op = CV.Rescale(rescale, shift)\n",
    "    hwc2chw_op = CV.HWC2CHW()\n",
    "    type_cast_op = C.TypeCast(mstype.int32)\n",
    "\n",
    "    # apply map operations on images\n",
    "    if not sparse:\n",
    "        one_hot_enco = C.OneHot(10)\n",
    "        ds1 = ds1.map(operations=one_hot_enco, input_columns=\"label\",\n",
    "                      num_parallel_workers=num_parallel_workers)\n",
    "        type_cast_op = C.TypeCast(mstype.float32)\n",
    "    ds1 = ds1.map(operations=type_cast_op, input_columns=\"label\",\n",
    "                  num_parallel_workers=num_parallel_workers)\n",
    "    ds1 = ds1.map(operations=resize_op, input_columns=\"image\",\n",
    "                  num_parallel_workers=num_parallel_workers)\n",
    "    ds1 = ds1.map(operations=rescale_op, input_columns=\"image\",\n",
    "                  num_parallel_workers=num_parallel_workers)\n",
    "    ds1 = ds1.map(operations=hwc2chw_op, input_columns=\"image\",\n",
    "                  num_parallel_workers=num_parallel_workers)\n",
    "\n",
    "    # apply DatasetOps\n",
    "    buffer_size = 10000\n",
    "    ds1 = ds1.shuffle(buffer_size=buffer_size)\n",
    "    ds1 = ds1.batch(batch_size, drop_remainder=True)\n",
    "    ds1 = ds1.repeat(repeat_size)\n",
    "\n",
    "    return ds1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 定义LeNet模型网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv(in_channels, out_channels, kernel_size, stride=1, padding=0):\n",
    "    weight = weight_variable()\n",
    "    return nn.Conv2d(in_channels, out_channels,\n",
    "                     kernel_size=kernel_size, stride=stride, padding=padding,\n",
    "                     weight_init=weight, has_bias=False, pad_mode=\"valid\")\n",
    "\n",
    "\n",
    "def fc_with_initialize(input_channels, out_channels):\n",
    "    weight = weight_variable()\n",
    "    bias = weight_variable()\n",
    "    return nn.Dense(input_channels, out_channels, weight, bias)\n",
    "\n",
    "\n",
    "def weight_variable():\n",
    "    return TruncatedNormal(0.02)\n",
    "\n",
    "\n",
    "class LeNet5(nn.Cell):\n",
    "    \"\"\"\n",
    "    Lenet network\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        super(LeNet5, self).__init__()\n",
    "        self.conv1 = conv(1, 6, 5)\n",
    "        self.conv2 = conv(6, 16, 5)\n",
    "        self.fc1 = fc_with_initialize(16*5*5, 120)\n",
    "        self.fc2 = fc_with_initialize(120, 84)\n",
    "        self.fc3 = fc_with_initialize(84, 10)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.flatten = nn.Flatten()\n",
    "\n",
    "    def construct(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 训练LeNet模型。利用上面定义的数据加载函数`generate_mnist_dataset`载入数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 step: 1875, loss is 0.5441832\n",
      "epoch: 2 step: 1875, loss is 0.18585104\n",
      "epoch: 3 step: 1875, loss is 0.27877027\n",
      "epoch: 4 step: 1875, loss is 0.01745773\n",
      "epoch: 5 step: 1875, loss is 0.025925232\n",
      "epoch: 6 step: 1875, loss is 0.033913765\n",
      "epoch: 7 step: 1875, loss is 0.06670261\n",
      "epoch: 8 step: 1875, loss is 0.0076941913\n",
      "epoch: 9 step: 1875, loss is 0.5493405\n",
      "epoch: 10 step: 1875, loss is 0.0064236256\n"
     ]
    }
   ],
   "source": [
    "# Downloading MNIST datasets from OBS.\n",
    "!mkdir -p ./datasets/MNIST_Data/train ./datasets/MNIST_Data/test\n",
    "!wget -NP ./datasets/MNIST_Data/train https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-labels-idx1-ubyte --no-check-certificate \n",
    "!wget -NP ./datasets/MNIST_Data/train https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-images-idx3-ubyte --no-check-certificate\n",
    "!wget -NP ./datasets/MNIST_Data/test https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-labels-idx1-ubyte --no-check-certificate\n",
    "!wget -NP ./datasets/MNIST_Data/test https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-images-idx3-ubyte --no-check-certificate\n",
    "\n",
    "mnist_path = \"./datasets/MNIST_Data/\"\n",
    "batch_size = 32\n",
    "# train original model\n",
    "ds_train = generate_mnist_dataset(os.path.join(mnist_path, \"train\"),\n",
    "                                  batch_size=batch_size, repeat_size=1,\n",
    "                                  sparse=False)\n",
    "\n",
    "train_images = []\n",
    "for data in ds_train.create_tuple_iterator():\n",
    "    images = data[0].asnumpy().astype(np.float32)\n",
    "    train_images.append(images)\n",
    "train_images = np.concatenate(train_images, axis=0)\n",
    "\n",
    "net = LeNet5()\n",
    "loss = SoftmaxCrossEntropyWithLogits(sparse=False)\n",
    "opt = nn.Momentum(net.trainable_params(), 0.01, 0.09)\n",
    "model = Model(net, loss, opt, metrics=None)\n",
    "model.train(10, ds_train, callbacks=[LossMonitor(1875)],\n",
    "            dataset_sink_mode=False)\n",
    "\n",
    "# get test data\n",
    "ds_test = generate_mnist_dataset(os.path.join(mnist_path, \"test\"),\n",
    "                                 batch_size=batch_size, repeat_size=1,\n",
    "                                 sparse=False)\n",
    "inputs = []\n",
    "labels = []\n",
    "for data in ds_test.create_tuple_iterator():\n",
    "    inputs.append(data[0].asnumpy().astype(np.float32))\n",
    "    labels.append(data[1].asnumpy())\n",
    "test_inputs = np.concatenate(inputs)\n",
    "test_labels = np.concatenate(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Fuzzer参数配置。  \n",
    "    设置数据变异方法及参数。支持同时配置多种方法，目前支持的数据变异方法包含三类：\n",
    "    - 图像仿射变换方法：Translate、Scale、Shear、Rotate。\n",
    "    - 基于图像像素值变化的方法： Contrast、Brightness、Blur、Noise。\n",
    "    - 基于对抗攻击的白盒、黑盒对抗样本生成方法：FGSM、PGD、MDIIM。  \n",
    "数据变异方法中一定要包含基于图像像素值变化的方法。\n",
    "\n",
    "前两种类型的图像变化方法，支持用户自定义配置参数，也支持算法随机选择参数。用户自定义参数配置范围请参考:https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/fuzz_testing/image_transform.py 中对应的类方法。算法随机选择参数，则params设置为'auto_param': [True]，参数将在推荐范围内随机生成。\n",
    "\n",
    "基于对抗攻击方法的参数配置请参考对应的攻击方法类。\n",
    "\n",
    "下面是变异方法及其参数配置的一个例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mutate_config = [{'method': 'Blur',\n",
    "                  'params': {'radius': [0.1, 0.2, 0.3],\n",
    "                             'auto_param': [True, False]}},\n",
    "                 {'method': 'Contrast',\n",
    "                  'params': {'auto_param': [True]}},\n",
    "                 {'method': 'Translate',\n",
    "                  'params': {'auto_param': [True]}},\n",
    "                 {'method': 'Brightness',\n",
    "                  'params': {'auto_param': [True]}},\n",
    "                 {'method': 'Noise',\n",
    "                  'params': {'auto_param': [True]}},\n",
    "                 {'method': 'Scale',\n",
    "                  'params': {'auto_param': [True]}},\n",
    "                 {'method': 'Shear',\n",
    "                  'params': {'auto_param': [True]}},\n",
    "                 {'method': 'FGSM',\n",
    "                  'params': {'eps': [0.3, 0.2, 0.4], 'alpha': [0.1]}}\n",
    "                ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "设置评价指标，目前支持5种评价指标，包括:\n",
    "\n",
    "- 通用评价指标：accuracy。\n",
    "- 神经元覆盖率指标：kmnc， nbc，snac。\n",
    "- 对抗攻击评价指标：attack_success_rate。 也可以设置为‘auto’，默认使用所有评价指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_metrics =['accuracy', 'kmnc', 'attack_success_rate']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 初始化种子队列，种子队列中的每个种子，包含2个值：原始图片、图片标签。这里取100个样本作为初始种子队列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make initial seeds\n",
    "initial_seeds = []\n",
    "for img, label in zip(test_inputs, test_labels):\n",
    "    initial_seeds.append([img, label])\n",
    "initial_seeds = initial_seeds[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. 测试Fuzz测试前的神经元覆盖率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] MA(5041:139824560817984,MainProcess):2021-02-09 17:46:41,372 [<ipython-input-9-b429a2d249a6>:5] [Fuzz_testing] KMNC of this test is : 0.0807\n"
     ]
    }
   ],
   "source": [
    "segmented_num=1000\n",
    "neuron_num=10\n",
    "model_coverage_test = ModelCoverageMetrics(model, neuron_num, segmented_num, train_images)\n",
    "model_coverage_test.calculate_coverage(np.array(test_inputs[:100]).astype(np.float32))\n",
    "LOGGER.info(TAG, 'KMNC of this test is : %s', model_coverage_test.get_kmnc())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果：\n",
    ">KMNC of this test is : 0.0807"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Fuzz测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_metrics = 'auto'\n",
    "model_fuzz_test = Fuzzer(model, train_images, neuron_num, segmented_num)\n",
    "_, _, _, _, metrics = model_fuzz_test.fuzzing(mutate_config, initial_seeds, eval_metrics=eval_metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. 实验结果  \n",
    "fuzzing的返回结果中包含了5个数据：fuzz生成的样本fuzz_samples、生成样本的真实标签true_labels、被测模型对于生成样本的预测值fuzz_preds、 生成样本使用的变异方法fuzz_strategies、fuzz testing的评估报告metrics_report。用户可使用这些返回结果进一步的分析模型的鲁棒性。这里只展开metrics_report，查看fuzz testing后的各个评估指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] MA(5041:139824560817984,MainProcess):2021-02-09 17:46:53,608 [<ipython-input-11-a30d8b9e535d>:3] [Fuzz_testing] Accuracy: 0.6404040404040404\n",
      "[INFO] MA(5041:139824560817984,MainProcess):2021-02-09 17:46:53,610 [<ipython-input-11-a30d8b9e535d>:3] [Fuzz_testing] Attack_success_rate: 0.3227091633466136\n",
      "[INFO] MA(5041:139824560817984,MainProcess):2021-02-09 17:46:53,610 [<ipython-input-11-a30d8b9e535d>:3] [Fuzz_testing] Neural_coverage_KMNC: 0.3714\n",
      "[INFO] MA(5041:139824560817984,MainProcess):2021-02-09 17:46:53,611 [<ipython-input-11-a30d8b9e535d>:3] [Fuzz_testing] Neural_coverage_NBC: 0.1\n",
      "[INFO] MA(5041:139824560817984,MainProcess):2021-02-09 17:46:53,612 [<ipython-input-11-a30d8b9e535d>:3] [Fuzz_testing] Neural_coverage_SNAC: 0.2\n"
     ]
    }
   ],
   "source": [
    "if metrics:\n",
    "    for key in metrics:\n",
    "        LOGGER.info(TAG, key + ': %s', metrics[key])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fuzz测试后结果如下：\n",
    ">Accuracy: 0.6404040404040404   \n",
    "Attack_success_rate: 0.3227091633466136  \n",
    "Neural_coverage_KMNC: 0.3714  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fuzz测试前种子的KMNC神经元覆盖率为8.1%，Fuzz后，KMNC神经元覆盖率为37.1%，神经元覆盖率提升，样本的多样性提升。Fuzz后，模型对于Fuzz生成样本的准确率为64%，使用了对抗攻击方法的样本，攻击成功率为32.27%。由于初始化种子、变异方法和相应的参数均为随机选择的，结果有一定的浮动是正常的。  \n",
    "\n",
    "原始图片：  \n",
    "![原始图片](https://gitee.com/mindspore/docs/raw/master/tutorials/training/source_zh_cn/advanced_use/images/fuzz_seed.png) \n",
    "\n",
    "\n",
    "Fuzz生成的变异图片：  \n",
    "![变异图片](https://gitee.com/mindspore/docs/raw/master/tutorials/training/source_zh_cn/advanced_use/images/fuzz_res.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
